Cosine Similarity Measure and Genetic Algorithm for extracting main content from web documents

Because of the use of growing information, web mining has become a primary necessity of world. Due to this, research on web mining has received a lot of interest from both industry and academia. Mining and prediction of user’s web browsing behaviors and deducing the actual content in a web document is one of the active subjects. The information on web is dirty. Apart from useful information, it contains unwanted information such as copyright notices and navigation bars that are not part of main contents of web pages. These seriously harm Web Data Mining and hence, need to be eliminated. This paper aims at studying the possible similarity criteria based on cosine similarity to deduce which parts of content are more important than others. Under vector space model, information retrieval is based on the similarity measurement between query and documents. Documents with high similarity to query are judge more relevant to the query and should be retrieved first. Under genetic algorithms, each query is represented by a chromosome. These chromosomes feed into genetic operator process: selection, crossover, and mutation until we get an optimize query chromosome for document retrieval. Our testing result show that information retrieval with 0.8 crossover probability and 0.01 mutation probability give the highest precision while 0.8 crossover probability and 0.3 mutation probability give the highest recall. KeywordsCosine Similarity; Genetic Algorithm; Fitness Function; Web Content Mining.